Skip to content

Conversation

@qinqon
Copy link
Contributor

@qinqon qinqon commented Nov 28, 2023

TODO:

  • Secondary networks need to have "route" fordward mode to be able to access ingress nodeport service at hosted cluster.

@openshift-ci openshift-ci bot requested review from csrwng and heliubj18 November 28, 2023 09:51
@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch 4 times, most recently from 94a5a1a to 6f708f7 Compare November 30, 2023 12:45
@qinqon
Copy link
Contributor Author

qinqon commented Nov 30, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-multinet-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 6f708f7 to 6213898 Compare November 30, 2023 12:49
@qinqon
Copy link
Contributor Author

qinqon commented Nov 30, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 6213898 to 4f94a8b Compare December 1, 2023 06:53
@qinqon
Copy link
Contributor Author

qinqon commented Dec 1, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 4f94a8b to 491153e Compare December 1, 2023 09:41
@qinqon
Copy link
Contributor Author

qinqon commented Dec 1, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch 2 times, most recently from f841e1d to bedf0b7 Compare December 1, 2023 11:47
@qinqon
Copy link
Contributor Author

qinqon commented Dec 1, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch 2 times, most recently from dddde1d to 65f9f0a Compare December 1, 2023 14:51
@qinqon
Copy link
Contributor Author

qinqon commented Dec 1, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 1, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: qinqon
Once this PR has been reviewed and has the lgtm label, please assign davidvossel for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@qinqon qinqon changed the title hypershift, kubevirt: Add periodic 4.15 multinet conformance hypershift, kubevirt: Add periodic 4.15 secondary-network conformance Dec 4, 2023
@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 65f9f0a to d770bd2 Compare December 4, 2023 06:09
@qinqon
Copy link
Contributor Author

qinqon commented Dec 4, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

1 similar comment
@qinqon
Copy link
Contributor Author

qinqon commented Dec 4, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from d770bd2 to 28fb18e Compare December 4, 2023 10:04
@qinqon
Copy link
Contributor Author

qinqon commented Dec 4, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 28fb18e to 4668502 Compare December 4, 2023 11:54
@qinqon
Copy link
Contributor Author

qinqon commented Dec 4, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

1 similar comment
@qinqon
Copy link
Contributor Author

qinqon commented Dec 4, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon
Copy link
Contributor Author

qinqon commented Dec 4, 2023

Transient error

Error: initializing source docker://quay.io/openshifttest/squid-proxy:multiarch: Requesting bearer token: invalid status code from registry 504 (Gateway Timeout)

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 2ddc1d8 to 0c518ad Compare December 5, 2023 05:39
@qinqon
Copy link
Contributor Author

qinqon commented Dec 5, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 0c518ad to 454dd05 Compare December 5, 2023 07:31
@qinqon
Copy link
Contributor Author

qinqon commented Dec 5, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon
Copy link
Contributor Author

qinqon commented Dec 5, 2023

As expected hosted cluster is not able to access the apiserver using the internal IP

[   29.411036] ignition[940]: GET error: Get "https://ignition-server-clusters-8002b878f9c9c715edc8.apps.ostest.test.metalkube.org/ignition": dial tcp 192.168.111.4:443: connect: network is unreachable^M
[   30.142770] NetworkManager[932]: <info>  [1701767056.4855] dhcp4 (enp2s0): state changed new lease, address=192.168.222.34^M
[   30.150357] NetworkManager[932]: <info>  [1701767056.4860] policy: set 'Wired Connection' (enp2s0) as default for IPv4 routing and DNS^M
[   30.529380] NetworkManager[932]: <info>  [1701767056.8723] dhcp4 (enp1s0): state changed new lease, address=192.168.221.33^M
[   30.538821] NetworkManager[932]: <info>  [1701767056.8732] policy: set 'Wired Connection' (enp1s0) as default for IPv4 routing and DNS^M
[   34.407634] ignition[940]: GET https://ignition-server-clusters-8002b878f9c9c715edc8.apps.ostest.test.metalkube.org/ignition: attempt #10^M
[   34.415137] ignition[940]: GET error: Get "https://ignition-server-clusters-8002b878f9c9c715edc8.apps.ostest.test.metalkube.org/ignition": dial tcp 192.168.111.4:443: connect: connection refused^M
[   39.412209] ignition[940]: GET https://ignition-server-clusters-8002b878f9c9c715edc8.apps.ostest.test.metalkube.org/ignition: attempt #11^M
[   39.427246] ignition[940]: GET error: Get "https://ignition-server-clusters-8002b878f9c9c715edc8.apps.ostest.test.metalkube.org/ignition": dial tcp 192.168.111.4:443: connect: connection refused^M

We can try to trait this cluster as external infra cluster or change libvirt network forward mode as route and configure hypervisor as such.

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 454dd05 to a46938f Compare December 5, 2023 11:49
@qinqon
Copy link
Contributor Author

qinqon commented Dec 5, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon
Copy link
Contributor Author

qinqon commented Dec 5, 2023

Running now with INSTALLER_PROXY and manually configing masquerade at the equinix hypervisor to have internet access.

@qinqon
Copy link
Contributor Author

qinqon commented Dec 5, 2023

Looks like is able to access ignition although the hostname is still wrong

Ignition: ran on 2023/12/05 13:21:45 UTC (at least 2 boots ago)
Ignition: user-provided config was applied
localhost login: 
Red Hat Enterprise Linux CoreOS 415.92.202311302101-0 (Plow) 4.15
SSH host key: SHA256:pD0P+2Lwbafcl1Tpu3s43MPdKzwMqs8hyzsxhlNMBSo (ED25519)
SSH host key: SHA256:92VvmDelC3zle97wS9UnZq9K9TznFWrcR871UO36YVM (ECDSA)
SSH host key: SHA256:GM8zsAENq0zlKjcujjhGLWoSD7/LUIiNiWnYqdH8rUs (RSA)
enp1s0:  
enp2s0: 192.168.222.24 fd2e:6f44:5dd8:cc56::33
Ignition: ran on 2023/12/05 13:21:45 UTC (at least 2 boots ago)
Ignition: user-provided config was applied
localhost login: [  342.442849] Warning: Deprecated Driver is detected: nft_compat will not be maintained in a future major release and may be disabled

@qinqon
Copy link
Contributor Author

qinqon commented Dec 5, 2023

Afterburn fix is still not at bootimage

[   50.908841] systemd[1]: Afterburn Hostname was skipped because no trigger condition checks were met.

@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from a46938f to 0abe8c8 Compare December 12, 2023 08:21
@qinqon
Copy link
Contributor Author

qinqon commented Dec 12, 2023

/pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance

@qinqon
Copy link
Contributor Author

qinqon commented Dec 12, 2023

hostname is correctly set now

[   43.909615] systemd[1]: Reached target Initrd Root File System.^M            
[   43.915260] systemd[1]: Starting Afterburn Hostname...^M                     
[   43.920224] systemd[1]: Starting Ignition OSTree: Check Root Filesystem Size...^M
[   43.926059] afterburn[1397]: Dec 12 09:59:32.253 INFO wrote hostname 4c619a37888adb5badd1-dc7426e2-7mlvd to /sysroot/etc/hostname^M
[   43.933766] systemd[1]: Starting Mount OSTree /var...^M                      
[   43.937947] ignition-ostree-mount-var[1404]: Mounting /sysroot/sysroot/ostree/deploy/rhcos/var^M
[   43.945808] systemd[1]: Finished Ignition OSTree: Check Root Filesystem Size.^M
[   43.952324] systemd[1]: Finished Mount OSTree /var.^M                        
[   43.957597] systemd[1]: Starting Ignition (mount)...^M                       
[   43.961886] systemd[1]: Finished Afterburn Hostname.^M                       

@qinqon
Copy link
Contributor Author

qinqon commented Dec 12, 2023

hosted cluster is a pretty good state missing part is the expected ingress

NAME                                       VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                    4.15.0-0.ci-2023-12-12-024252   False       True          False      38m     DeploymentAvailable: 0 replicas available for console deployment...
csi-snapshot-controller                    4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
dns                                        4.15.0-0.ci-2023-12-12-024252   True        False         False      37m     
image-registry                             4.15.0-0.ci-2023-12-12-024252   True        False         False      38m     
ingress                                    4.15.0-0.ci-2023-12-12-024252   True        False         True       37m     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
insights                                   4.15.0-0.ci-2023-12-12-024252   True        False         False      38m     
kube-apiserver                             4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
kube-controller-manager                    4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
kube-scheduler                             4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
kube-storage-version-migrator              4.15.0-0.ci-2023-12-12-024252   True        False         False      38m     
monitoring                                 4.15.0-0.ci-2023-12-12-024252   True        False         False      36m     
network                                    4.15.0-0.ci-2023-12-12-024252   True        False         False      39m     
node-tuning                                4.15.0-0.ci-2023-12-12-024252   True        False         False      40m     
openshift-apiserver                        4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
openshift-controller-manager               4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
openshift-samples                          4.15.0-0.ci-2023-12-12-024252   True        False         False      33m     
operator-lifecycle-manager                 4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
operator-lifecycle-manager-catalog         4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
operator-lifecycle-manager-packageserver   4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     
service-ca                                 4.15.0-0.ci-2023-12-12-024252   True        False         False      38m     
storage                                    4.15.0-0.ci-2023-12-12-024252   True        False         False      45m     

We need to configure an external ingress since serices for secondary interfaces do not exists so the out of the box ingress solution will not work.

@openshift-bot
Copy link
Contributor

Issues in openshift/release go stale after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 15d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 18, 2024
@qinqon qinqon changed the title hypershift, kubevirt: Add periodic 4.15 secondary-network conformance hypershift, kubevirt: Add periodic secondary-network conformance Feb 2, 2024
@qinqon qinqon force-pushed the hypershift-kubevirt-multinet-conformance branch from 0abe8c8 to 21438dc Compare February 5, 2024 07:36
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@qinqon: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-cluster-network-operator-master-e2e-aws-hypershift-ovn-kubevirt openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.17-e2e-aws-hypershift-ovn-kubevirt openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.16-e2e-aws-hypershift-ovn-kubevirt openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.15-e2e-aws-hypershift-ovn-kubevirt openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.14-e2e-aws-hypershift-ovn-kubevirt openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-hypershift-main-e2e-kubevirt-aws-ovn openshift/hypershift presubmit Registry content changed
pull-ci-openshift-hypershift-release-4.17-e2e-kubevirt-aws-ovn openshift/hypershift presubmit Registry content changed
pull-ci-openshift-hypershift-release-4.16-e2e-kubevirt-aws-ovn openshift/hypershift presubmit Registry content changed
pull-ci-openshift-hypershift-release-4.15-e2e-kubevirt-aws-ovn openshift/hypershift presubmit Registry content changed
pull-ci-openshift-hypershift-release-4.14-e2e-kubevirt-aws-ovn openshift/hypershift presubmit Registry content changed
pull-ci-openshift-hypershift-release-4.13-e2e-kubevirt-aws-ovn openshift/hypershift presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-kubevirt openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.17-e2e-aws-ovn-kubevirt openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.16-e2e-aws-ovn-kubevirt openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.15-e2e-aws-ovn-kubevirt openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.14-e2e-aws-ovn-kubevirt openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-cloud-provider-kubevirt-main-e2e-kubevirt-aws-ovn openshift/cloud-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.17-e2e-kubevirt-aws-ovn openshift/cloud-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.16-e2e-kubevirt-aws-ovn openshift/cloud-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.15-e2e-kubevirt-aws-ovn openshift/cloud-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.14-e2e-kubevirt-aws-ovn openshift/cloud-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-main-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.17-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.16-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.15-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed

A total of 57 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@qinqon
Copy link
Contributor Author

qinqon commented Feb 5, 2024

/pj-rehearse

@openshift-bot
Copy link
Contributor

Stale issue in openshift/release rot after 15d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 15d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 20, 2024
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 1, 2024

@qinqon: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-multinet-conformance 6f708f781185e07cab6acbb5a05e472ec75caf2a link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-multinet-conformance
ci/rehearse/periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance 0abe8c8 link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance
ci/rehearse/periodic-ci-openshift-hypershift-release-4.16-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance 21438dc link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-4.16-periodics-e2e-kubevirt-baremetalds-secondary-network-conformance
ci/rehearse/periodic-ci-openshift-hypershift-release-4.13-periodics-e2e-kubevirt-mce-baremetalds-conformance 21438dc link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-4.13-periodics-e2e-kubevirt-mce-baremetalds-conformance
ci/rehearse/periodic-ci-openshift-hypershift-release-4.13-periodics-e2e-kubevirt-mce-conformance 21438dc link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-4.13-periodics-e2e-kubevirt-mce-conformance
ci/rehearse/openshift/cluster-api-provider-kubevirt/release-4.17/e2e-hypershift-kubevirt 21438dc link unknown /pj-rehearse pull-ci-openshift-cluster-api-provider-kubevirt-release-4.17-e2e-hypershift-kubevirt
ci/prow/ci-operator-config 21438dc link true /test ci-operator-config

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2024
@openshift-merge-robot
Copy link
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot
Copy link
Contributor

Rotten issues in openshift/release close after 15d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci openshift-ci bot closed this Mar 17, 2024
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 17, 2024

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues in openshift/release close after 15d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants